Does ANN returns the same prediction for the same input?

Question

Does ANN predictable? By this I mean that if I re-run the same script over and over, does it make sense that the error (MAE / MSE / R^2) is different on every run?

if true, then a follow-up question: is it a good practice to run the script several times and save the NN that produce the lowest error?

My script:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Standardize the data
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Define the model with a Masking layer
model = Sequential()
model.add(Masking(mask_value=0., input_shape=(X_train_scaled.shape[1],)))
model.add(Dense(64, activation='relu'))
model.add(Dense(42, activation='relu'))
model.add(Dense(1, activation='linear'))  # Linear activation for regression
model.summary()

# Compile the model
model.compile(optimizer='adam', loss='mean_absolute_error')

# Fit the model to your data
model.fit(X_train_scaled, y_train, epochs=50, batch_size=42, validation_data=(X_test_scaled, y_test))


#  Evaluate the model
y_pred = model.predict(X_test_scaled)

# Calculate evaluation metrics
mae_nn = metrics.mean_absolute_error(y_test, y_pred)
mse_nn = metrics.mean_squared_error(y_test, y_pred)
rmse_nn = np.sqrt(mse_nn)
r2_nn = metrics.r2_score(y_test, y_pred)
explained_variance_nn = metrics.explained_variance_score(y_test, y_pred)

# Print the summary of evaluation metrics
print(f"MAE: {round(mae_nn, 3)}, RMSE: {round(rmse_nn, 3)}, R^2: {round(r2_nn, 3)}, Explained Variance Score: {round(explained_variance_nn, 3)}")

Answer 1

The training (i.e. model.fit) of your neural network is NOT deterministic → every time you train the network, the result of the training (i.e. the model) is different. The inference (i.e. model.predict) of your neural network IS deterministic → every time you request predictions from a specific model you obtain the same result.

Your training is not deterministic due to 2 reasons:

• Weight initialization: layer weights are initialized to random numbers.
• Data shuffling: data is shuffled randomly when preparing the batches to feed the model.

You can train your model multiple times and get the one with best performance on the validation data, but this does not assure you that it will be the best performing on unseen data, which is the important thing when training a model. Nevertheless, it is common to do it.

As a piece of advice, I would say that it would be great if you could replicate your own training, should the need arise. For this, you can check how to do it as decribed in the Keras docs, which basically tells you to do as follows:

keras.utils.set_random_seed(812)

# If using TensorFlow, this will make GPU ops as deterministic as possible,
# but it will affect the overall performance, so be mindful of that.
tf.config.experimental.enable_op_determinism()

Different values for the random seed will give you different trained models. If you want to make your results reproducible, you should store somewhere the random seed that gave you the best results.

Finally, note that, while your specific neural network is deterministic at inference time, this does not apply to all neural networks.