Deep learning model produces very different results when classifying the same samples

Question

I'm trying to design a simple deep learning application for biometric system verification, but every time I run the application I get very different results and I can't figure out why. I don't use data augmentation in the training and development sets to get close results, but the results are still very different. The dataset I am using is a biometric verification set with about 10 seconds of video from each sample. I have tried many methods such as sample reduction, synthetic sample creation, data shuffle, but the results have not changed. You can check the Train Loss values in Results1. Besides, threshold is the value obtained from the FAR-FRR curve of the development set and is equal to the EER. ACCURACY (Threshold) is used to manually assign classes that are above and below the threshold value (EER and HTER are biometric validation metrics and are obtained from the development set and test set respectively. These two values are expected to be close to zero). Also, the validation_loss value sometimes increases and sometimes decreases instead of decreasing smoothly. Although I use earlystopping, I cannot correct the loss value. I would like you to help me with this issue.

Note: The model I am using is an example from the Keras documentation (link is in the code). I use a for loop to run the program multiple times. Each iteration is one round of the for loop, each Run is a manual execution of the program. Also, when model = make_model is taken out of the for loop, I get very different results (Results2-Make Model out of the for loop). I don't understand the reason for this either. Could this be caused by the randomization of the initial weights? Do you have any suggestions for a solution?

The code example I used is as follows;

The Definitions:

import numpy as np
import os
import keras.preprocessing.image
import keras.callbacks
import tensorflow as tf

tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)

from tensorflow.keras import layers
from tensorflow import keras
from sklearn.metrics import confusion_matrix

os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
tf.keras.utils.set_random_seed(42)

from sklearn.metrics import roc_curve

def biometric_metrics_fixed(y_true, scores):
    fpr, tpr, threshold = roc_curve(y_true, scores, pos_label=1)
    fnr = 1 - tpr
    eer_threshold = threshold[np.nanargmin(np.absolute((fnr - fpr)))]
    eer = fpr[np.nanargmin(np.absolute((fnr - fpr)))]
    return eer, fpr, fnr, eer_threshold, tpr

def Metrics(y_test, y_test_pred, th, eer, train_acc, test_acc, epoch):
    from sklearn import metrics
    y_test_pred_th = np.where(y_test_pred < th, 0, 1)
    tn, fp, fn, tp = confusion_matrix(y_test, y_test_pred_th).ravel()
    fpr = fp / (fp + tn)
    fnr = fn / (fn + tp)

    x = {
        "ACC": metrics.accuracy_score(y_test, y_test_pred_th),
        "AUROC": metrics.roc_auc_score(y_test, y_test_pred_th),
        "EER": eer,
        "F1": metrics.f1_score(y_test, y_test_pred_th),
        "HTER": ((fpr + fnr) / 2),
        "MAE": metrics.mean_absolute_error(y_test, y_test_pred_th),
        "MSE": metrics.mean_squared_error(y_test, y_test_pred_th),
        "PREC": metrics.precision_score(y_test, y_test_pred_th),
        "RECALL": metrics.recall_score(y_test, y_test_pred_th),
        "TRAIN_ACC": train_acc,
        "TEST_ACC": test_acc,
        "XX_EPOCH": epoch
    }
    
    return x

physical_devices = tf.config.experimental.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], True)

image_size = (128, 128)
batch_size = 128
IMG_SHAPE = image_size + (3,)

all_results = {"All":{}}

base_dir = "All"

# Keras example: https://keras.io/examples/vision/image_classification_from_scratch/
def make_model(input_shape, num_classes):
    inputs = keras.Input(shape=input_shape)

    # Entry block
    x = layers.Rescaling(1.0 / 255)(inputs)
    x = layers.Conv2D(128, 3, strides=2, padding="same")(x)
    x = layers.BatchNormalization()(x)
    x = layers.Activation("relu")(x)
    
    previous_block_activation = x  # Set aside residual
    
    for size in [256, 512, 728]:
        x = layers.Activation("relu")(x)
        x = layers.SeparableConv2D(size, 3, padding="same")(x)
        x = layers.BatchNormalization()(x)
    
        x = layers.Activation("relu")(x)
        x = layers.SeparableConv2D(size, 3, padding="same")(x)
        x = layers.BatchNormalization()(x)
    
        x = layers.MaxPooling2D(3, strides=2, padding="same")(x)
    
        # Project residual
        residual = layers.Conv2D(size, 1, strides=2, padding="same")(
            previous_block_activation
        )
        x = layers.add([x, residual])  # Add back residual
        previous_block_activation = x  # Set aside next residual
    
    x = layers.SeparableConv2D(1024, 3, padding="same")(x)
    x = layers.BatchNormalization()(x)
    x = layers.Activation("relu")(x)
    
    x = layers.GlobalAveragePooling2D()(x)
    if num_classes == 2:
        activation = "sigmoid"
        units = 1
    else:
        activation = "softmax"
        units = num_classes
    
    x = layers.Dropout(0.5)(x)
    outputs = layers.Dense(units, activation=activation)(x)
    return keras.Model(inputs, outputs)

Data Loading:

folder_name = "Database_Folder"

folder = os.path.join(folder_name, base_dir)

train_ds = tf.keras.utils.image_dataset_from_directory(
os.path.join(folder,"train"),
image_size=image_size,
batch_size=batch_size,
)

val_ds = tf.keras.utils.image_dataset_from_directory(
os.path.join(folder,"devel"),
image_size=image_size,
batch_size=batch_size,
)

test_ds = tf.keras.utils.image_dataset_from_directory(
os.path.join(folder,"test"),
image_size=image_size,
batch_size=batch_size,
)

# Prefetching samples in GPU memory helps maximize GPU utilization.
train_ds = train_ds.prefetch(tf.data.AUTOTUNE)
val_ds = val_ds.prefetch(tf.data.AUTOTUNE)
test_ds = test_ds.prefetch(tf.data.AUTOTUNE)

epochs = 100

train_ds_list = list(train_ds)
x_train = np.concatenate([x for x, y in train_ds_list], axis=0)
y_train = np.concatenate([y for x, y in train_ds_list], axis=0)

val_ds_list = list(val_ds)
x_val = np.concatenate([x for x, y in val_ds_list], axis=0)
y_val = np.concatenate([y for x, y in val_ds_list], axis=0)

test_ds_list = list(test_ds)
x_test = np.concatenate([x for x, y in test_ds_list], axis=0)
y_test = np.concatenate([y for x, y in test_ds_list], axis=0)

with tf.device('/cpu:0'):
x_train = tf.convert_to_tensor(x_train, np.float32)
y_train = tf.convert_to_tensor(y_train, np.float32)

with tf.device('/cpu:0'):
x_val = tf.convert_to_tensor(x_val, np.float32)
y_val = tf.convert_to_tensor(y_val, np.float32)

with tf.device('/cpu:0'):
x_test = tf.convert_to_tensor(x_test, np.float32)
y_test = tf.convert_to_tensor(y_test, np.float32)

Classification multiple times:

for i in range(5):
    keras.backend.clear_session()
    model = make_model(input_shape=image_size + (3,), num_classes=2)
    
    model.compile(
        optimizer=keras.optimizers.Adam(1e-3),
        loss="binary_crossentropy",
        metrics=["accuracy"],
    )
    
    model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(
        filepath=os.path.join("Checkpoints", folder_name + "_" + str(i+1) + "_model_save_at_{epoch}.h5"),
        monitor='val_loss',
        mode='min',
        restore_best_weights=True,
        save_best_only=True)
    
    callbacks = [
        keras.callbacks.EarlyStopping(
            monitor='val_loss', mode='min', verbose=1, patience=10),
        model_checkpoint_callback
    ]
        
    history = model.fit(
        x_train,
        y_train,
        epochs=epochs,
        callbacks=callbacks,
        batch_size=batch_size,
        validation_data=(x_val, y_val),
    )
        
    y_val_pred = model.predict(x_val)
    eer, far, frr, th, tpr = biometric_metrics_fixed(y_val, y_val_pred)
    
    y_test_pred = model.predict(x_test)
    
    test_loss, test_accuracy = model.evaluate(x_test, y_test)
    
    all_results[base_dir][i] = Metrics(y_test, y_test_pred, th, eer, history.history['accuracy'][-1], test_accuracy, len(history.history['accuracy']))
    all_results[base_dir][i]["XXX_History"] = history.history
    all_results[base_dir][i]["XXX_TestLoss"] = test_loss

Here are the results I got when I ran this application 3 times;

If I take the following codes outside the for loop, this time the results are as follows;

keras.backend.clear_session()

model = make_model(input_shape=image_size + (3,), num_classes=2)

In this case the results are very close to each other, but I am concerned about whether the model is affected by the previous loop state.

Answer 1

You're facing a reproducibility issue, I think.

In the first case (having clear_session and make_model within the loss) you get different results at each training run because the model, and the data loading pipeline as well (in case you shuffle samples), depend on random numbers that are NOT fixed (i.e. not deterministic.)

This means that each time you instantiate a model, with make_model(...) the weights are initialized in a different way, and also the way the batches are formed is different.

In the second case, instead, where you create the model outside the loop you are training it multiple times. in fact, from the time you can see that the metrics improve over the runs.

Now, if your aim is to measure the variability of each training run, you should pick a sequence of random seeds (one per loop iteration), and set all the random generators to that seed. For example:

random_seeds = [1, 2, 3, 4, 5]

for i, seed in enumerate(random_seeds):
  fix_random_generators(seed)  # <-- you should implement this
  
  # this should be not required
  # keras.backend.clear_session()
  model = make_model(input_shape=image_size + (3,), num_classes=2)

  # ...

Then, once you have all models trained you evaluate on the test set and compare the metrics to, for example, measure the variance/uncertainty of them.