Deep learning test loss curve won't go down

Question

I've been working with Deep Learning projects for this current project that I am working on and it's basically a time series classification problem. Where given an array of time series data I need to classify the customers as being either honest or dishonest.

The current model that I have now only uses CNNs, but I'm planning on expanding it with LSTMs or other models in the future. Here is the code for my model.

model = Sequential([
    Input(batch_input_shape = (None, 1036, 1)),

    Conv1D(
        filters=32,
        kernel_size=3,
        padding='same',
        activation='relu',
        activity_regularizer=l2(5e-4),
    ),
    Conv1D(
        filters=16,
        kernel_size=3,
        padding='same',
        activation='relu',
        activity_regularizer=l2(5e-4),
    ),
    MaxPooling1D(),
    
    Conv1D(
        filters=8,
        kernel_size=3,
        padding='same',
        activation='relu',
        activity_regularizer=l2(5e-4),
    ),
    Conv1D(
        filters=8,
        kernel_size=3,
        padding='same',
        activation='relu',
        activity_regularizer=l2(5e-4),
    ),
    MaxPooling1D(),
    Flatten(),

    Dense(10, activation='relu'),
    Dropout(0.25),
    Dense(1, activation='sigmoid'),
])

model.compile(
    # optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4),
    loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
    metrics=[f1_m,precision_m, recall_m, matthews_correlation, 'accuracy', fpr_m]
)

After training the model for 100 EPOCHs the loss curve is this

I've tried a number of things which are:

• Reducing the network size. This still results in the same issue but at a different point in the network
• Reducing the learning rate of the optimizer. Same with the point above. This seems to work but it is in fact changing when this happens.

I am open to any input or recommendations I could get on how to make the testing curve follow the training curve more. I should note that my dataset is imbalanced and I'm ONLY balancing the x_train and y_train using SMOTE and not making and balancing to the testing and validation datasets to keep the data as clean as possible.

Answer 1

Your learning curves suggest overfitting.

In this respect you should try

• increasing dropout
• adjust l1/l2 regularisation to higher levels
• inspect class ratios in training and validation tests ("learning on easy predicting on difficult" imbalances)

Additionally, you may want to

• add batchnorm layer
• try different activations
• check to what extend weights/bias of each layer get updated during training iterations, in case of vahishing/exploding gradients