0
$\begingroup$

I have a binary classification problem. I get the following results: a val_loss (far) lower than the train_loss, but the accuracy is also lower for the validation compared to the training set. How is that possible ?

Epoch 1/10
10708/10708 [=======] - loss: 0.6356 - acc: 0.8289 - val_loss: 0.4981 - val_acc: 0.7760
Epoch 2/10
10708/10708 [=======] - loss: 0.6243 - acc: 0.8248 - val_loss: 0.5075 - val_acc: 0.7609
Epoch 3/10
10708/10708 [=======] - loss: 0.6204 - acc: 0.8302 - val_loss: 0.5152 - val_acc: 0.7694
Epoch 4/10
10708/10708 [=======] - loss: 0.6215 - acc: 0.8307 - val_loss: 0.4981 - val_acc: 0.7824
Epoch 5/10
10708/10708 [=======] - loss: 0.6180 - acc: 0.8318 - val_loss: 0.4942 - val_acc: 0.7848

This is the Keras model I'm using:

model = Sequential()
model.add(LSTM(16, input_shape=(1, 1)))
model.add(Dense(2, activation='softmax'))

adam = Adam(lr=0.001)
model.compile(loss='categorical_crossentropy',
          optimizer=adam,
          metrics=['accuracy'])
$\endgroup$
2
  • $\begingroup$ Are you using mini-batches when you fit the model? $\endgroup$
    – Tophat
    Jan 3, 2018 at 21:59
  • $\begingroup$ I am using a generator with model.fit_generator(). My batches are also not of the same size $\endgroup$
    – vvvvv
    Jan 3, 2018 at 22:01

2 Answers 2

2
$\begingroup$

How did you split your training and test sets? My guess is that the minority class is under-represented in your test set. Consider the case where the minority class has frequencies of 25% and 5% in the training and test sets, respectively. If my model predicts everything is the majority class, the recall for the minority class is 0% on both sets but we get 75% training accuracy and 95% validation accuracy. Now let's say we train the model a bit more and achieve 50% recall while still correctly labeling all of the majority class samples: now we're up to 87% training and 97% accuracy. Because the training and test sets exhibit significantly different class frequencies, the test set will basically always have a higher accuracy than the training set (up to the point where you overfit, at which point the validation accuracy could be be lower than training accuracy but not necessarily).

The solution here is to either use stratified sampling to make sure your classes have equal representation between your different datasets, or to monitor other evaluation metrics like precision and recall. Plotting the taining and validation accuracies could also be helpful, since it should be visually pretty obvious if your model starts to overfit, regardless of class imbalance between your training and test sets.

$\endgroup$
4
  • $\begingroup$ yes, but what is strange to me is not the fact the validation accuracy is lower than the training accuracy. It's the strange fact that the loss is also lower $\endgroup$
    – vvvvv
    Jan 3, 2018 at 23:11
  • 1
    $\begingroup$ I'm pretty sure the phenomenon I'm describing accounts for that as well. Have you checked on the class proportions? That would be a pretty simple way to determine whether or not I'm on to something. $\endgroup$
    – David Marx
    Jan 3, 2018 at 23:14
  • $\begingroup$ Sorry. Yes the data is unbalanced in general (80%/20%). I did not check whether the proportions still apply the same way in the training set and the test set but I think they do since I shuffle $\endgroup$
    – vvvvv
    Jan 3, 2018 at 23:22
  • 1
    $\begingroup$ I don't know if you're waiting for me to beg or what, but if you're concerned enough to post here about it I'd strongly recommend you actually take the second to check on those proportions rather than just assuming that they're the same. $\endgroup$
    – David Marx
    Jan 4, 2018 at 3:50
0
$\begingroup$

This is a common thing with neural networks and different batch sizes. The training loss is the average of losses for the minibatch. Naturally for the first few batches you'll have a higher loss and as it goes through the data the loss gets smaller. Mean while the loss for the validation set is calculated against the entire dataset.

$\endgroup$
1
  • $\begingroup$ Are you sure you don't have that backwards? $\endgroup$
    – D.W.
    Apr 4, 2018 at 20:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.