Accuracy vs Categorical Accuracy

Question

I was running a DNN model that uses ResNet50 for Transfer Learning. While fitting the training data on my model to check the initial trend (would run for more epochs if initial trend seems right), I noticed that even though my training ACCURACY metric consistently remained 0.0000e+00 for 3 epochs, my training CATEGORICAL ACCURACY metric increased from 0.4689 to 0.5278. The loss showed a decreasing trend as well. Similar behaviour was observed for validation set.

Code Snippet:

from tensorflow.keras.applications.resnet50 import ResNet50
METRICS = [
    tf.keras.metrics.Accuracy(),
    tf.keras.metrics.CategoricalAccuracy(),
]

tf.keras.backend.clear_session()
model = tf.keras.Sequential()

pretrained_model = ResNet50(
    include_top = False,
    weights = "imagenet",
    input_shape = (512,512,3),
    pooling = "avg"
)
for layer in pretrained_model.layers:
  layer.trainable = False

model.add(pretrained_model)
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(512,activation="relu"))
model.add(tf.keras.layers.Dense(3,activation="softmax"))

CLASS_WEIGHTS = get_CW_dict(distribution)

model.compile(
    optimizer = tf.keras.optimizers.SGD(),
    loss = tf.keras.losses.CategoricalCrossentropy(),
    metrics = METRICS, steps_per_execution=639
    )

history = model.fit(
    x = TRAIN_GEN,
    validation_data = VALID_GEN,
    epochs = 5, batch_size=8,
    callbacks = [tf.keras.callbacks.TensorBoard(log_dir="logs/fit")],
    class_weight = CLASS_WEIGHTS
)

OUTPUT:

Epoch 1/5
160/160 [==============================] - 107s 669ms/step - loss: 0.4983 - accuracy: 0.0000e+00 - categorical_accuracy: 0.4689 - val_loss: 1.1232 - val_accuracy: 0.0000e+00 - val_categorical_accuracy: 0.4891
Epoch 2/5
160/160 [==============================] - 99s 616ms/step - loss: 0.3922 - accuracy: 0.0000e+00 - categorical_accuracy: 0.5088 - val_loss: 1.1451 - val_accuracy: 0.0000e+00 - val_categorical_accuracy: 0.5031
Epoch 3/5
160/160 [==============================] - 98s 614ms/step - loss: 0.3853 - accuracy: 0.0000e+00 - categorical_accuracy: 0.5278 - val_loss: 1.1407 - val_accuracy: 0.0000e+00 - val_categorical_accuracy: 0.5094

Now, my questions are:

1. Should I consider Categorical Accuracy and ignore 'Accuracy' metric in this case?

2. Considering that TF/Keras automatically chooses the accuracy metric on the basis of the activation function of the output layer and the type of loss function, what may be the reason for such ambiguous behavior?

Answer 1

Investigating the Keras docs, there is a major difference between Accuracy and categorical_accuracy:

Accuracy:

This metric creates two local variables, total and count that are used to compute the frequency with which y_pred matches y_true. This frequency is ultimately returned as binary accuracy: an idempotent operation that simply divides total by count.

categorical_accuracy:

This metric creates two local variables, total and count that are used to compute the frequency with which y_pred matches y_true. This frequency is ultimately returned as categorical accuracy: an idempotent operation that simply divides total by count. y_pred and y_true should be passed in as vectors of probabilities, rather than as labels. If necessary, use tf.one_hot to expand y_true as a vector.

And since you're using Categorical Cross Entropy for your loss (assuming you have converted your labels to one hot encoded labels) then you should use categorical accuracy for your accuracy metric.