Validation loss increases and validation accuracy decreases

Question

I have an issue with my model. I'm trying to use the most basic Conv1D model to analyze review data and output a rating of 1-5 class, therefore the loss is categorical_crossentropy. Model structure is as below

# define model
model = Sequential()
model.add(Embedding(vocab_size, 100, input_length=max_length))
model.add(Conv1D(filters=32, kernel_size=8, activation='relu'))
model.add(MaxPooling1D(pool_size=2))
model.add(Flatten())
model.add(Dense(10, activation='relu'))
model.add(Dense(5, activation='softmax'))

# compile network
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

# fit network
model.fit(final_X_train, final_Y_train, epochs=5, batch_size=50, validation_data=(final_X_val, final_Y_val), callbacks=callback)

Total params: 14,977,717

Trainable params: 14,977,717

Non-trainable params: 0

Train on 212135 samples, validate on 69472 samples

Epoch 1/5
loss: 0.9452 - acc: 0.5781 - val_loss: 0.8440 - val_acc: 0.6309

Epoch 2/5
loss: 0.7707 - acc: 0.6711 - val_loss: 0.8234 - val_acc: 0.6433

Epoch 3/5
loss: 0.5807 - acc: 0.7657 - val_loss: 0.9144 - val_acc: 0.6345

Epoch 4/5
loss: 0.3736 - acc: 0.8575 - val_loss: 1.1982 - val_acc: 0.6194

Epoch 5/5
loss: 0.2285 - acc: 0.9173 - val_loss: 1.5770 - val_acc: 0.6073

Training acc increases and loss decreases as expected. But validation loss and validation acc decrease straight after the 2nd epoch itself. The overall testing after training gives an accuracy around 60s.

The total accuracy is : 0.6046845041714888

I've already cleaned, shuffled, down-sampled (all classes have 42427 number of data samples) and split the data properly to training(70%) / validation(10%) / testing(20%).

If you see any improvements to fix this problem, please let me know. :)

Answer 1

What you are experiencing is known as overfitting, and it’s a common problem in machine learning and data science.

Overfitting happens when a model begins to focus on the noise in the training data set and extracts features based on it. This helps the model to improve its performance on the training set but hurts its ability to generalize so the accuracy on the validation set decreases.

To deal with overfitting, you need to use regularization during the training. You can try adding dropout layers or batch-normalization layers, adding weight regularization, or you can artificially increase the size of your training set by performing some data augmentation.

You can read more about it in the following post:

What are the possible approaches to fixing Overfitting on a CNN?