2
$\begingroup$

I've been working with Deep Learning projects for this current project that I am working on and it's basically a time series classification problem. Where given an array of time series data I need to classify the customers as being either honest or dishonest.

The current model that I have now only uses CNNs, but I'm planning on expanding it with LSTMs or other models in the future. Here is the code for my model.

model = Sequential([
    Input(batch_input_shape = (None, 1036, 1)),

    Conv1D(
        filters=32,
        kernel_size=3,
        padding='same',
        activation='relu',
        activity_regularizer=l2(5e-4),
    ),
    Conv1D(
        filters=16,
        kernel_size=3,
        padding='same',
        activation='relu',
        activity_regularizer=l2(5e-4),
    ),
    MaxPooling1D(),
    
    Conv1D(
        filters=8,
        kernel_size=3,
        padding='same',
        activation='relu',
        activity_regularizer=l2(5e-4),
    ),
    Conv1D(
        filters=8,
        kernel_size=3,
        padding='same',
        activation='relu',
        activity_regularizer=l2(5e-4),
    ),
    MaxPooling1D(),
    Flatten(),

    Dense(10, activation='relu'),
    Dropout(0.25),
    Dense(1, activation='sigmoid'),
])

model.compile(
    # optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4),
    loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
    metrics=[f1_m,precision_m, recall_m, matthews_correlation, 'accuracy', fpr_m]
)

After training the model for 100 EPOCHs the loss curve is this enter image description here

I've tried a number of things which are:

  • Reducing the network size. This still results in the same issue but at a different point in the network
  • Reducing the learning rate of the optimizer. Same with the point above. This seems to work but it is in fact changing when this happens.

I am open to any input or recommendations I could get on how to make the testing curve follow the training curve more. I should note that my dataset is imbalanced and I'm ONLY balancing the x_train and y_train using SMOTE and not making and balancing to the testing and validation datasets to keep the data as clean as possible.

$\endgroup$
3
  • $\begingroup$ Just an idea but did you try without using SMOTE, i.e. using only the real data? (or using SMOTE less). It's clear that the model starts overfitting around epoch 10, it might be due to the fact there's too little real data in the minority class so the model starts using the minor variations generated by SMOTE as clues, thus causing the overfitting. This is just an hypothesis of course. $\endgroup$ – Erwan Jan 26 at 14:44
  • $\begingroup$ @Erwan Thank you for your response. I've tried removing SMOTE and training the model without it but I'm getting an almost identical graph to the one in the original post. I've also taken some time to revise the code and it seems to me like all of my data preprocessing and everything else before the training stage are correct. $\endgroup$ – Joseph Anderson Jan 26 at 14:58
  • $\begingroup$ Ok then, I can't really think of anything else... There must be something in the data which causes it, but it's hard to see what. I would try a few things like reducing the training set size and/or undersampling the majority class, just to try to get an idea of what happens. $\endgroup$ – Erwan Jan 27 at 10:25
0
$\begingroup$

Your learning curves suggest overfitting.

In this respect you should try

  • increasing dropout
  • adjust l1/l2 regularisation to higher levels
  • inspect class ratios in training and validation tests ("learning on easy predicting on difficult" imbalances)

Additionally, you may want to

  • add batchnorm layer
  • try different activations
  • check to what extend weights/bias of each layer get updated during training iterations, in case of vahishing/exploding gradients
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.