I have Keras neural network for binary classification with final layer having one output with Sigmoid activation. I have noticed that large amount of output numbers are strictly one or zero (rather than between 0 and 1 as expected). What could be the reason for this?
At first I thought maybe network is 100% sure about accuracy of these numbers, but noticed that some of these predictions are actually incorrect.
model = tf.keras.models.Sequential() model.add(tf.keras.layers.Dense(32, activation = tf.nn.relu, input_shape=(X_train.shape,))) model.add(tf.keras.layers.Dense(64, activation = tf.nn.relu)) model.add(tf.keras.layers.Dense(32, activation = tf.nn.relu)) model.add(tf.keras.layers.Dense(1, activation = tf.nn.sigmoid)) model.compile(optimizer = tf.keras.optimizers.Adam(), loss = 'binary_crossentropy', metrics = ['accuracy', roc_auc]) model.fit(X_train, y_train, epochs=10, validation_data=(X_test, y_test)) y_pred = model.predict(X_test) yhat =  for pred in y_pred: if pred >= 0.7: yhat.append(1) else: yhat.append(0)
Data consists of 8 columns (features) and 270 000 rows (9th column is 'y' column). Out of these 270,000 rows only 9% contained labels of class 1 (rest were zero), so I downsampled the data (just removed bunch of data with label 0), trained the model and then did the prediction on full data. I modified how ones and zeros were determined by Sigmoid though, I changed threshold from 0.5 to 0.7 (which was the ROC score I got on downsampled data)