Why are the accuracy and loss graphs of my Keras binary model not smooth?

Question

I am creating a binary model and using TensorBoard to visualize the graph of the accuracy and loss. However, I noticed that the graphs for my model's accuracy and loss are not smooth. Why is that?

Here are my model's accuracy and loss graphs:

Accuracy:

Loss:

As you can see, the graphs above are not smooth. Here is my code for debugging:

import keras
from keras import models, layers, regularizers, optimizers, callbacks

def create_model(input_shape):
    model = models.Sequential([
        layers.Input(shape=input_shape),
        
        layers.Dense(256, kernel_regularizer=regularizers.l2(0.002)),
        layers.BatchNormalization(),
        layers.Activation("relu"),
        layers.Dropout(0.4),
        
        layers.Dense(512, kernel_regularizer=regularizers.l2(0.002)),
        layers.BatchNormalization(),
        layers.Activation("relu"),
        layers.Dropout(0.4),
        
        layers.Dense(512, kernel_regularizer=regularizers.l2(0.002)),
        layers.BatchNormalization(),
        layers.Activation("relu"),
        layers.Dropout(0.4),
        
        layers.Dense(256, kernel_regularizer=regularizers.l2(0.002)),
        layers.BatchNormalization(),
        layers.Activation("relu"),
        layers.Dropout(0.4),
        
        layers.Dense(128, kernel_regularizer=regularizers.l2(0.002)),
        layers.BatchNormalization(),
        layers.Activation("relu"),
        layers.Dropout(0.4),
        
        layers.Dense(1, activation="sigmoid")
    ])
    return model

input_shape = x_train.shape[1:]
model = create_model(input_shape)

optimizer = optimizers.AdamW(learning_rate=0.001)

model.compile(
    optimizer=optimizer,
    loss="binary_crossentropy",
    metrics=["accuracy", keras.metrics.AUC()]
)

reduce_lr = callbacks.ReduceLROnPlateau(monitor="val_loss", factor=0.2, patience=3, min_lr=1e-6, verbose=1)
model_checkpoint = callbacks.ModelCheckpoint("best_model.keras", monitor="val_loss", mode="min", save_best_only=True, verbose=1)
tensorboard = callbacks.TensorBoard(log_dir="./logs")

data_augmentation = keras.Sequential([
    layers.GaussianNoise(0.1),
])

history = model.fit(
    data_augmentation(x_train),
    y_train,
    batch_size=64,
    epochs=500,
    validation_split=0.2,
    callbacks=[reduce_lr, model_checkpoint, tensorboard],
    class_weight={0: 1, 1: 1},
    shuffle=True
)

test_loss, test_accuracy, test_auc = model.evaluate(x_test, y_test)
print(f"Test loss: {test_loss:.4f}")
print(f"Test accuracy: {test_accuracy:.4f}")
print(f"Test AUC: {test_auc:.4f}")

Please note that the shape of x_train is (121, 1), and the shape of y_train is (121,).

Answer 1

A combination of:

• Learning is inherently stochastic.
• In your case you add noise.
• Accuracy is not a very good and stable metric itself (as you can see the loss is relatively smooth). This is particularly true for small / imbalanced data sets.

Answer 2

Most likely explanation:

There are data points that the model is learning very strongly, but some data points are causing the model to struggle around 0.5 threshold.

For example, if a prediction moves up from 55% to 90%, this will reduce the loss, but it won't necessarily impact accuracy.

What you can do:

Look at the data points where the model is struggling and work on fixing them e.g Outliers