I have a problem. I would like to solve a NLP classification problem. For this I have trained a CNN and since I have other features, I wanted to include them in the model training. Thus I have concatenated a CNN and the other features.

However, the problem is that the loss jumps to 175,10,100.8,... . The val_loss, on the other hand, is at 0, something. What is the reason that the loss is so high and erratic?

Is it because the features and CNN cannot interpret the model correctly? Should the features be trained by a standalone model first?

What is the reason that the model has such an erratic loss? And what does that tell us?

enter image description here

enter image description here

class CNN_1D:
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def forward(self):
            # filter_sizes = [1,2,3,5]
            # num_filters = 32
            extra_nb_features = df_train.shape[1]

            inp = Input(shape=(maxlen, ))
            extra_inp = Input(shape=(extra_nb_features, ))

            x = Embedding(embedding_matrix.shape[0], 300, weights=[embedding_matrix], trainable=False)(inp)
            x = SpatialDropout1D(0.4)(x)
            # x = Reshape((maxlen, embed_size, 1))(x)

            x = Conv1D(256, 7, activation='relu')(x)
            x = MaxPooling1D()(x)
            x = Conv1D(128, 5, activation="relu")(x)
            x = MaxPooling1D()(x)
            x = Dropout(0.2)(x)  
            #x = Flatten()(x)
            x = GlobalMaxPooling1D()(x)
            combined = Concatenate(axis=-1)([x, extra_inp])

            combined = Dropout(0.15)(combined)
            outp = Dense(128, activation="relu")(combined)

            outp = Dense(numbmer, activation="softmax")(outp)

            model = Model(inputs=[inp, extra_inp] , outputs=outp)
            return model
  • $\begingroup$ Are you sure the accuracy graph is correctly labelled? Because the val accuracy is higher than training accuracy, this is definitely not normal. If the graph is correct, I don't even know what can cause this but I would guess something is wrong in the data. What is the size of the data? Number of instances by class? Also I'd suggest starting with a traditional method like decision tree: this would give you a baseline perf and you could check more easily if the data/evaluation part works correctly. $\endgroup$
    – Erwan
    Commented Oct 25, 2022 at 14:33
  • $\begingroup$ Very small batch_size can cause this. Also, add your pre-processing steps and model training parms. $\endgroup$
    – 10xAI
    Commented Oct 25, 2022 at 15:13


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.