I have Keras neural network for binary classification with final layer having one output with Sigmoid activation. I have noticed that large amount of output numbers are strictly one or zero (rather than between 0 and 1 as expected). What could be the reason for this?

At first I thought maybe network is 100% sure about accuracy of these numbers, but noticed that some of these predictions are actually incorrect.



model = tf.keras.models.Sequential()

model.add(tf.keras.layers.Dense(32, activation = tf.nn.relu, input_shape=(X_train.shape[1],)))

model.add(tf.keras.layers.Dense(64, activation = tf.nn.relu))

model.add(tf.keras.layers.Dense(32, activation = tf.nn.relu))

model.add(tf.keras.layers.Dense(1, activation = tf.nn.sigmoid))

model.compile(optimizer = tf.keras.optimizers.Adam(), loss = 'binary_crossentropy', metrics = ['accuracy', roc_auc])

model.fit(X_train, y_train, epochs=10, validation_data=(X_test, y_test))

y_pred = model.predict(X_test)

yhat = []

for pred in y_pred:
    if pred >= 0.7:

Data consists of 8 columns (features) and 270 000 rows (9th column is 'y' column). Out of these 270,000 rows only 9% contained labels of class 1 (rest were zero), so I downsampled the data (just removed bunch of data with label 0), trained the model and then did the prediction on full data. I modified how ones and zeros were determined by Sigmoid though, I changed threshold from 0.5 to 0.7 (which was the ROC score I got on downsampled data)

  • $\begingroup$ Can you share some more detail on your network structure and the data? $\endgroup$ – serali Nov 15 '19 at 10:37
  • $\begingroup$ @serali I have edited the question $\endgroup$ – Ach113 Nov 15 '19 at 10:44
  • $\begingroup$ Hi, please avoid posting images of code / outputs which can be pasted as text in the question. $\endgroup$ – Romain Reboulleau Nov 15 '19 at 10:58
  • $\begingroup$ @Ach113 Could you share the actual code you use to build the network and the way you check the values of the final sigmoid? Its probably a small mistake in the code, it is impossible to tell by looking just at the summary. $\endgroup$ – serali Nov 15 '19 at 11:27
  • $\begingroup$ @serali Sure, updated the post $\endgroup$ – Ach113 Nov 15 '19 at 11:31

Apparently the problem was that I forgot to normalize data. As I found out, sigmoid turns any input greater than 10 into 1 and inputs less than -10 into 0. Normalization solves this problem (as normalized values are between 0 and 1)

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.