Keras 1D CNN always predicts the same result even if accuracy is high on training set

Question

The validation accuracy of my 1D CNN is stuck on 0.5 and that's because I'm always getting the same prediction out of a balanced data set. At the same time my training accuracy keeps increasing and the loss decreasing as intended.

Strangely, if I do model.evaluate() on my training set (that has close to 1 accuracy in the last epoch), the accuracy will also be 0.5. How can the accuracy here differ so much from the training accuracy of the last epoch? I've also tried with a batch size of 1 for both training and evaluating and the problem persists.

Well, I've been searching for different solutions for quite some time but still no luck. Possible problems I've already looked into:

1. My data set is properly balanced and shuffled;
2. My labels are correct;
3. Tried adding fully connected layers;
4. Tried adding/removing dropout from the fully connected layers;
5. Tried the same architecture, but with the last layer with 1 neuron and sigmoid activation;
6. Tried changing the learning rates (went down to 0.0001 but still the same problem).

---

Here's my code:

import pathlib
import numpy as np
import ipynb.fs.defs.preprocessDataset as preprocessDataset
import pickle
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras import Input
from tensorflow.keras.layers import Conv1D, BatchNormalization, Activation, MaxPooling1D, Flatten, Dropout, Dense
from tensorflow.keras.optimizers import SGD

main_folder = pathlib.Path.cwd().parent
datasetsFolder=f'{main_folder}\\datasets'
trainDataset = preprocessDataset.loadDataset('DatasetTime_Sg12p5_Ov75_Train',datasetsFolder)
testDataset = preprocessDataset.loadDataset('DatasetTime_Sg12p5_Ov75_Test',datasetsFolder)

X_train,Y_train,Names_train=trainDataset[0],trainDataset[1],trainDataset[2]
X_test,Y_test,Names_test=testDataset[0],testDataset[1],testDataset[2]

model = Sequential()

model.add(Input(shape=X_train.shape[1:]))

model.add(Conv1D(16, 61, strides=1, padding="same"))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling1D(2, strides=2, padding="valid"))

model.add(Conv1D(32, 3, strides=1, padding="same"))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling1D(2, strides=2, padding="valid"))

model.add(Conv1D(64, 3, strides=1, padding="same"))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling1D(2, strides=2, padding="valid"))

model.add(Conv1D(64, 3, strides=1, padding="same"))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling1D(2, strides=2, padding="valid"))

model.add(Conv1D(64, 3, strides=1, padding="same"))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Flatten())
model.add(Dropout(0.5))

model.add(Dense(200))
model.add(Activation('relu'))

model.add(Dense(2))
model.add(Activation('softmax'))

opt = SGD(learning_rate=0.01)

model.compile(loss='binary_crossentropy',optimizer=opt,metrics=['accuracy'])

model.summary()

model.fit(X_train,Y_train,epochs=10,shuffle=False,validation_data=(X_test, Y_test))

model.evaluate(X_train,Y_train)

---

Here's model.fit():

model.fit(X_train,Y_train,epochs=10,shuffle=False,validation_data=(X_test, Y_test))

Epoch 1/10
914/914 [==============================] - 277s 300ms/step - loss: 0.6405 - accuracy: 0.6543 - val_loss: 7.9835 - val_accuracy: 0.5000
Epoch 2/10
914/914 [==============================] - 270s 295ms/step - loss: 0.3997 - accuracy: 0.8204 - val_loss: 19.8981 - val_accuracy: 0.5000
Epoch 3/10
914/914 [==============================] - 273s 298ms/step - loss: 0.2976 - accuracy: 0.8730 - val_loss: 1.9558 - val_accuracy: 0.5002
Epoch 4/10
914/914 [==============================] - 278s 304ms/step - loss: 0.2897 - accuracy: 0.8776 - val_loss: 20.2678 - val_accuracy: 0.5000
Epoch 5/10
914/914 [==============================] - 277s 303ms/step - loss: 0.2459 - accuracy: 0.8991 - val_loss: 5.4945 - val_accuracy: 0.5000
Epoch 6/10
914/914 [==============================] - 268s 294ms/step - loss: 0.2008 - accuracy: 0.9181 - val_loss: 32.4579 - val_accuracy: 0.5000
Epoch 7/10
914/914 [==============================] - 271s 297ms/step - loss: 0.1695 - accuracy: 0.9317 - val_loss: 14.9538 - val_accuracy: 0.5000
Epoch 8/10
914/914 [==============================] - 276s 302ms/step - loss: 0.1423 - accuracy: 0.9452 - val_loss: 1.4420 - val_accuracy: 0.4988
Epoch 9/10
914/914 [==============================] - 266s 291ms/step - loss: 0.1261 - accuracy: 0.9497 - val_loss: 4.3830 - val_accuracy: 0.5005
Epoch 10/10
914/914 [==============================] - 272s 297ms/step - loss: 0.1142 - accuracy: 0.9548 - val_loss: 1.6054 - val_accuracy: 0.5009

Here's model.evaluate():

model.evaluate(X_train,Y_train)

914/914 [==============================] - 35s 37ms/step - loss: 1.7588 - accuracy: 0.5009

Here's model.summary():

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv1d (Conv1D)              (None, 4096, 16)          992       
_________________________________________________________________
batch_normalization (BatchNo (None, 4096, 16)          64        
_________________________________________________________________
activation (Activation)      (None, 4096, 16)          0         
_________________________________________________________________
max_pooling1d (MaxPooling1D) (None, 2048, 16)          0         
_________________________________________________________________
conv1d_1 (Conv1D)            (None, 2048, 32)          1568      
_________________________________________________________________
batch_normalization_1 (Batch (None, 2048, 32)          128       
_________________________________________________________________
activation_1 (Activation)    (None, 2048, 32)          0         
_________________________________________________________________
max_pooling1d_1 (MaxPooling1 (None, 1024, 32)          0         
_________________________________________________________________
conv1d_2 (Conv1D)            (None, 1024, 64)          6208      
_________________________________________________________________
batch_normalization_2 (Batch (None, 1024, 64)          256       
_________________________________________________________________
activation_2 (Activation)    (None, 1024, 64)          0         
_________________________________________________________________
max_pooling1d_2 (MaxPooling1 (None, 512, 64)           0         
_________________________________________________________________
conv1d_3 (Conv1D)            (None, 512, 64)           12352     
_________________________________________________________________
batch_normalization_3 (Batch (None, 512, 64)           256       
_________________________________________________________________
activation_3 (Activation)    (None, 512, 64)           0         
_________________________________________________________________
max_pooling1d_3 (MaxPooling1 (None, 256, 64)           0         
_________________________________________________________________
conv1d_4 (Conv1D)            (None, 256, 64)           12352     
_________________________________________________________________
batch_normalization_4 (Batch (None, 256, 64)           256       
_________________________________________________________________
activation_4 (Activation)    (None, 256, 64)           0         
_________________________________________________________________
flatten (Flatten)            (None, 16384)             0         
_________________________________________________________________
dropout (Dropout)            (None, 16384)             0         
_________________________________________________________________
dense (Dense)                (None, 200)               3277000   
_________________________________________________________________
activation_5 (Activation)    (None, 200)               0         
_________________________________________________________________
dense_1 (Dense)              (None, 2)                 402       
_________________________________________________________________
activation_6 (Activation)    (None, 2)                 0         
=================================================================
Total params: 3,311,834
Trainable params: 3,311,354
Non-trainable params: 480
_________________________________________________________________

Here are the first 5 rows of X_train and Y_train:

[[ 3.602187e-04]
 [ 8.075248e-04]
 [ 4.319834e-04]
 ...
 [ 3.011377e-05]
 [-1.693150e-04]
 [-8.542318e-05]] [0. 1.]

[[ 2.884359e-04]
 [-6.340756e-05]
 [-5.905452e-06]
 ...
 [-9.305983e-05]
 [ 1.345304e-04]
 [-1.366256e-04]] [0. 1.]

[[ 7.720405e-04]
 [ 6.031118e-05]
 [ 6.691568e-04]
 ...
 [-6.443140e-05]
 [ 1.998355e-04]
 [ 5.839724e-05]] [1. 0.]

[[-3.294961e-04]
 [ 6.234528e-05]
 [-2.861797e-04]
 ...
 [-4.983012e-04]
 [ 3.897884e-04]
 [-1.014846e-05]] [0. 1.]

[[-0.0001479 ]
 [ 0.00037975]
 [-0.00024007]
 ...
 [ 0.00018743]
 [ 0.00044564]
 [-0.00025613]] [0. 1.]

Answer 1

• The binary cross-entropy loss function is based on the assumption that there is only one output node and it can have a value between 0-1.

• If you have more than two outputs and using the softmax activation function, you should use a categorical cross-entropy loss function in order to handle the multiclass situation.

• But in your scenario, there is a binary classification problem, so you need to have just 1 output node, and the last activation function needs to be a sigmoid function for adjusting the output between 0 and 1.

    y_train = y_train[:,0]
    y_test = y_test[:,0]
  

And model should be like:

    model.add(Dense(200))
    model.add(Activation('relu'))    
    model.add(Dense(20))
    model.add(Activation('relu')) 
    model.add(Dense(1))
    model.add(Activation('sigmoid'))    
    opt = SGD(learning_rate=0.01)    
    model.compile(loss='binary_crossentropy',optimizer=opt,metrics=['accuracy'])

Answer 2

The solution for my problem was implementing Batch Renormalization: BatchNormalization(renorm=True). In addition normalizing the inputs helped a lot improving the overall performance of the neural network.