CNN always predicts either 0 or 1 for binary classification

Question

I am using a Kaggle dataset on stress characteristics, derived from ECG signals, and I would like to train a CNN to recognize stress/non-stress situations.

I have built a model in Keras:

model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(U1, (3, 1), activation = 'relu', input_shape = (num_features, 1, 1)),
    tf.keras.layers.Conv2D(U2, (3, 1), activation = 'relu'),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(U3, activation = 'relu'),
    tf.keras.layers.Dense(1,  activation = 'sigmoid')
])

where U1, U2 and U3 are the parameters I have been changing to find the right combination to ensure the best performance.

What I did is, specifically:

1. divide the samples in training and test set (I don't have a validation set as the number of available samples is small);
2. normalize both training and validation set by dividing them by the max value found in the samples;
3. train the network with various combinations of U1, U2 and U3 to find the ones ensuring the best performance.

The training is done as follows:

model.compile(loss = 'binary_crossentropy' , optimizer = 'adam' , metrics = ['accuracy'])
history = model.fit(x_train, y_train,
          epochs = 50,
          batch_size = 160,
          validation_data = (x_test, y_test))
score = model.evaluate(x_test, y_test, batch_size=128)

The network performs really well on both the training and test set, achieving an accuracy of 99.3% on test set (and 99.5% on training set).

However, when applied on real data (by taking one's ECG, computing the features and normalizing them by the same normalization value used on training and test set above), the network is always predicting:

• a label of 0.0 for "normal" ECGs;
• a label of 1.0 for noisy ECGs (which are taken as stressed ECGs).

It bugs me that no other labels rather than 0.0 or 1.0 are never returned. It is true that sometimes the network predicts labels of exactly 0.0 or exactly 1.0 also for samples in the training and validation set, but never seeing a label different from 0.0 and 1.0 in real-world data sounds strange.

Is this a problem of the network? Or a problem of the dataset? Maybe could this be related to the fact that the real-world ECG data I am using are not extracted from the same distribution of the Kaggle dataset? I see that, for instance, real-world data have profound differences in values with respect to the data in the Kaggle dataset (so much difference that even after normalization values are not really normalized), but I don't know if this is a valid reason that justifies the problem I see.

Answer 1

Sigmoid functions might have saturation problems. The values that it receives are probably too far from zero, and the sigmoid is returning 'extreme' results (i.e.: 0 or 1). I suggest you to keep your signal zero centered by putting BatchNormalization() layers between each of your layers (certainly between Dense() ones, but also between conv layers if you prefer).

Additionally, sigmoid functions are thought to perform not very well in general. The main configuration for classifiers is to put one output node for each target category (in this case, 2) and use softmax activation. Your loss at this point would be BinaryCrossentropy(). This requires one-hot encoding of your target variable.

I would also try other regularization tricks, such as dropout and l1-l2 regularization, or different initializations. There no rule of thumb here, you have to play with these hyperparameters and figure out what's the best configuration for your task.