8
$\begingroup$

I am training a CNN on some text data. The sentences are padded and embedded and fed to a CNN. The model architecture is:

model = Sequential()
model.add(Embedding(max_features, embedding_dims, input_length=maxlen))
model.add(Conv1D(128, 5, activation='relu'))
model.add(GlobalMaxPooling1D())

model.add(Dense(50, activation = 'relu'))
model.add(BatchNormalization())

model.add(Dense(50, activation = 'relu'))
model.add(BatchNormalization())

model.add(Dense(25, activation = 'relu'))
#model.add(Dropout(0.2))
model.add(BatchNormalization())

model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

enter image description here

Any help would be appreciated.

$\endgroup$
3
$\begingroup$

Your model is completely overfitting. The training loss is constantly decreasing but the validation loss isn't. This means that the your current model is complex enough to 'memorize' the patterns in the training data. In such situations, you need to regularize your model.

To regularize your data you may try any or all combination of the following:

1) Reduce the layers of the neural network.

2) Reduce the number of neurons in each layer of the network to reduce the number of parameters.

3) Add dropout and tune its rate.

4) Use L2 normalisation on the parameter weights and tune the lambda value.

5) If possible add more data for training.

6) Adding more data may not always be economically viable option so 'Data Augmentation' can play a big role here. Data Augmentation depends on the type of input to the neural nets. For images they comprise of rotation, translation, affine transformation like shear etc.

$\endgroup$
1
  • 1
    $\begingroup$ For text data, a commonly used (with great impact) method of reducing overfitting by bringing in extra data is using word (or subword) embeddings that are pre-trained on a very large unlabeled corpus. Word2vec, fasttext, BERT and friends, etc are all variations on that concept. One could even argue that replacing words with pre-trained embeddings is something like the linguistic equivalent of the image augmentation methods. $\endgroup$
    – Peteris
    Dec 27 '19 at 2:26
2
$\begingroup$

You are over-fitting to the train dataset and failing to generalize to the validation dataset.

One way to reduce overfitting is regularization. The most common methods for regularizing a neural network are:

  1. Add more data.
  2. Add dropout.
  3. Reduce the number of layers.
$\endgroup$
1
  • $\begingroup$ There's dropout in the model in the question. Should OP add more dropout steps then or change the dropout rate? $\endgroup$
    – NelsonGon
    Jun 26 '20 at 14:54
0
$\begingroup$

Also a couple of things that proved valuable besides mentioned are batch normalisation and feewer deep layers. The less complexity the more generalisation power you can have.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.