1
$\begingroup$

Both models are for binary classification problems

Model 1

tf.random.set_seed(42)

model_1 = tf.keras.Sequential([
    tf.keras.layers.Dense(4, activation='relu'),
    tf.keras.layers.Dense(4, activation='relu'),
    tf.keras.layers.Dense(1)
])


model_1.compile(loss='binary_crossentropy',
                optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
                metrics=['accuracy'])


history = model_1.fit(X, y, epochs=250)

Model 2

tf.random.set_seed(42)

model_2 = tf.keras.Sequential([
    tf.keras.layers.Dense(4, activation=tf.keras.activations.relu),
    tf.keras.layers.Dense(4, activation=tf.keras.activations.relu),
    tf.keras.layers.Dense(1)
])


model_2.compile(loss='binary_crossentropy',
                optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
                metrics=['accuracy'])


history = model_2.fit(X, y, epochs=250)

Both models perform differently, model 1 performs with 0.5 accuracy and model 2 performs with 0.85-0.9ish accuracy. I'm confused why it's different. For model 1, I'd say the recent why it's performing bad is because I didn't define the output activation function. However, why does model 2 perform well despite not including the output activation function?

$\endgroup$
3
  • 1
    $\begingroup$ Are those accuracies consistent across multiple trainings of the same model? (note that tf.random.set_seed is not enough to ensure reproducibility, see this; instead, consider using tf.keras.utils.set_random_seed) $\endgroup$
    – noe
    Nov 14, 2023 at 7:23
  • 1
    $\begingroup$ Yeah it seems that it's not consistently poor, I just got unlucky and tested model 1 a few times and got all 0.5ish and model 2 a few times and only got a few bad ones. After setting the seed, the results are exactly the same $\endgroup$ Nov 14, 2023 at 9:10
  • 1
    $\begingroup$ I added an answer so that this question can be marked as resolved. $\endgroup$
    – noe
    Nov 14, 2023 at 17:16

1 Answer 1

2
$\begingroup$

Your results were probably result of the randomness in the training. Both activation=tf.keras.activations.relu and activation='relu' are equivalent.

To obtain reproducible results, consider using tf.keras.utils.set_random_seed instead of tf.random.set_seed, because the latter is not enough (see this).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.