0
$\begingroup$

I am using a Kaggle dataset on stress characteristics, derived from ECG signals, and I would like to train a CNN to recognize stress/non-stress situations.

I have built a model in Keras:

model = tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(U1, (3, 1), activation = 'relu', input_shape = (num_features, 1, 1)),
    tf.keras.layers.Conv2D(U2, (3, 1), activation = 'relu'),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(U3, activation = 'relu'),
    tf.keras.layers.Dense(1,  activation = 'sigmoid')
])

where U1, U2 and U3 are the parameters I have been changing to find the right combination to ensure the best performance.

What I did is, specifically:

  1. divide the samples in training and test set (I don't have a validation set as the number of available samples is small);
  2. normalize both training and validation set by dividing them by the max value found in the samples;
  3. train the network with various combinations of U1, U2 and U3 to find the ones ensuring the best performance.

The training is done as follows:

model.compile(loss = 'binary_crossentropy' , optimizer = 'adam' , metrics = ['accuracy'])
history = model.fit(x_train, y_train,
          epochs = 50,
          batch_size = 160,
          validation_data = (x_test, y_test))
score = model.evaluate(x_test, y_test, batch_size=128)

The network performs really well on both the training and test set, achieving an accuracy of 99.3% on test set (and 99.5% on training set).

However, when applied on real data (by taking one's ECG, computing the features and normalizing them by the same normalization value used on training and test set above), the network is always predicting:

  • a label of 0.0 for "normal" ECGs;
  • a label of 1.0 for noisy ECGs (which are taken as stressed ECGs).

It bugs me that no other labels rather than 0.0 or 1.0 are never returned. It is true that sometimes the network predicts labels of exactly 0.0 or exactly 1.0 also for samples in the training and validation set, but never seeing a label different from 0.0 and 1.0 in real-world data sounds strange.

Is this a problem of the network? Or a problem of the dataset? Maybe could this be related to the fact that the real-world ECG data I am using are not extracted from the same distribution of the Kaggle dataset? I see that, for instance, real-world data have profound differences in values with respect to the data in the Kaggle dataset (so much difference that even after normalization values are not really normalized), but I don't know if this is a valid reason that justifies the problem I see.

$\endgroup$
  • $\begingroup$ Are you working with image data? Are you sure that Conv layers are the right choice for that dataset? $\endgroup$ – Leevo Oct 2 '19 at 8:43
  • $\begingroup$ I followed some works in the literature, e.g.,: He et al., Real-Time Detection of Acute Cognitive Stress using a Convolutional Neural Network from Electrocardiographic Signal, IEEE Access 2019 $\endgroup$ – Eleanore Oct 2 '19 at 9:22
  • $\begingroup$ Sometimes, 1D convolutions are used for time series data. 2D convolutions can work only with 2D inputs (such as pixels). What kind of convolution did they employ? $\endgroup$ – Leevo Oct 2 '19 at 10:23
  • $\begingroup$ The model is pretty similar to mine, with the only difference that their feature vector is larger. Their input size is indeed 1D (799x1x1), they use 4x1x1 filters with stride 1x1. As I mentioned in the question, the network is performing in a good way on both training and test set. However, it seems that probabilities on real data (i.e., taking my own ECG and using it as an input) are always exactly 1.0 or exactly 0.0, and I would like to understand if this is a problem of the network or of the data that is not coherent with the training set. $\endgroup$ – Eleanore Oct 2 '19 at 10:33
0
$\begingroup$

Sigmoid functions might have saturation problems. The values that it receives are probably too far from zero, and the sigmoid is returning 'extreme' results (i.e.: 0 or 1). I suggest you to keep your signal zero centered by putting BatchNormalization() layers between each of your layers (certainly between Dense() ones, but also between conv layers if you prefer).

Additionally, sigmoid functions are thought to perform not very well in general. The main configuration for classifiers is to put one output node for each target category (in this case, 2) and use softmax activation. Your loss at this point would be BinaryCrossentropy(). This requires one-hot encoding of your target variable.

I would also try other regularization tricks, such as dropout and l1-l2 regularization, or different initializations. There no rule of thumb here, you have to play with these hyperparameters and figure out what's the best configuration for your task.

| improve this answer | |
$\endgroup$
  • $\begingroup$ so it is "natural" that I see this problem only on fresh data that are neither in the training nor in the validation set? could this be related to the fact that they are statistically different (e.g., the same that happens with photos taken with a smartphone vs professional photos)? $\endgroup$ – Eleanore Oct 2 '19 at 13:03
  • $\begingroup$ Make sure that you scaled the new data the same way you scaled the training set. Other than that, I don't have any experience with biomedical data. $\endgroup$ – Leevo Oct 2 '19 at 13:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.